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(57) ABSTRACT 

An image processing system for high performance digital 
imaging in a digital camera. The reflected light from an 
image is focused through a lens and optically filtered. A 
CCD array converts this image into an electrical signal. This 
electrical signal is processed and then converted into an 
equivalent digital signal. A digital signal processor is then 
used to process the raw digital signal. The DSP includes a 
capture data path, a data flow control, an image processing 
data path, a compression/decompression engine, a resize 
circuit, a display processing circuit, and a rotation circuit. 
Data is routed between the DSP and memory via a bus. By 
selectively activating and reusing certain parts of the hard- 
ware architecture and various data paths, at least four modes 
of operation can be supported: live view, instant review, and 
play mode. Furthermore, the correct image is automatically 
displayed in all four modes, regardless of the orientation of 
the image or the physical orientation of the camera (both at 
the time the picture was taken and at the time the picture is 
being rendered for display). 

18 Claims, 13 Drawing Sheets 
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IMAGE PROCESSING SYSTEM FOR HIGH 
PERFORMANCE DIGITAL IMAGING 
DEVICES 



FIELD OF THE INVENTION 

The present invention relates generally to digital cameras, 
and more particularly to an image processing system for 
high performance digital imaging. 

BACKGROUND OF THE INVENTION 

Given the rapid advances made in semiconductor tech- 
nology and digital imaging, it was just a matter of time 
before digital cameras were developed and introduced. Most 
digital cameras today are similar in size with and behave 
similar to conventional photographic cameras. But instead 
of capturing an image through a lens and recorded onto a 
photosensitive material as with traditional cameras, digital 
cameras capture an image using a charge-coupled device 
(CCD). The composite image is represented by thousands of 
discrete picture elements, known as pixels. The color of each 
pixel is given in the form of digital data consisting of binary 
encoded l's and O's. This digital data is then processed and 
stored into memory (e.g., internal flash memory, external 
memory cards, buffers, etc.) for subsequent use. 

By adopting a digital imaging approach, digital cameras 
offer several advantages over more traditional cameras 
because the bits of data can now be more readily manipu- 
lated using digital signal processing techniques. For 
instance, the backside of a digital cameras can be equipped 
with a liquid crystal display (LCD) screen. In a record mode 
of operation, the LCD acts as a viewfinder in which the LCD 
displays objects and scenes in real-time. At any time, the 
user can click to capture the "picture" displayed on the LCD. 
In a playback mode of operation, pictures can be displayed 
on the LCD either individually or in groups of four, nine, or 
sixteen pictures. Users can manually page through the 
current archive of stored pictures by pressing the appropriate 
buttons. Furthermore, a user can instantly delete certain 
pictures. In addition, pictures can be viewed in full size, 
scaled down in size, or zoomed to larger sizes. Another 
benefit is that digital cameras can rotate the orientation of the 
pictures so as to automatically switch between landscape 
versus portrait formats. A common feature found in many 
digital cameras is a burst mode of operation, whereby a 
single click of a button can cause the camera to take several 
pictures in rapid succession, thereby creating a film-like 
sequence of images. 

Due to their digital nature and in order to offer these 
advanced features, modern digital cameras were typically 
designed using specialized circuitry to handle each of the 
advanced features. Although this approach makes digital 
cameras relatively fast, it dramatically increases the overall 
cost, size, battery consumption, and weight of the digital 
cameras. In addition, the captured image display might 
appear to be slightly different than a live-view display if one 
were to use different circuits and data paths. 

One solution to these drawbacks is to couple a generic 
processor to memory via a bus, where the processor is 
programmed to perform all of the enhanced functionality's, 
and digital data is shuttled between the memory and pro- 
cessor via the bus. However, this approach suffers from one 
major drawback; it is relatively slow. When a user presses a 
button, he or she expects instant feedback and response. The 
user does not want to wait the number of seconds it takes to 
process an image. It has been discovered that the software 
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approach is slow because it primarily suffers from two 
disadvantages. First, because the advanced features are 
typically performed by software, it can take a long time to 
configure and execute the requisite programming steps. 

5 Second, the bus speed becomes a limiting factor as large 
amounts of data need to be transferred between the proces- 
sor and memory. Essentially, the bus becomes overloaded. 
The end result is that software-based digital cameras may be 
too slow to suit the tastes of consumers, depending on the 

10 processor/bus combination. 

Thus, there is a need for an advanced architecture which 
facilitates high-speed image processing, offers advanced 
features, and yet is cost-effective. The present invention 
provides a novel solution by implementing a special hard- 

15 ware configuration which has been optimized for increased 
speed. Certain parts and paths of the circuit are reused and 
shared so as to leverage existing resources with minimal 
impact on its speed and functionality. By sharing certain 
resources, duplication is reduced, thereby decreasing costs. 

20 Furthermore, the present invention is adaptable to be used in 
virtually any type of digital camera and CCD array. 

SUMMARY OF THE INVENTION 

The present invention pertains to an image processing 

25 system for high performance digital imaging in digital 
cameras and the like. The reflected light from an image is 
focused through a lens and optically filtered. A CCD array 
converts this image into an electrical signal. This electrical 
signal is processed and then converted into an equivalent 

30 digital signal. A digital signal processor (DSP) is then used 
to process the raw digital signal. The DSP includes a capture 
data path, a data flow control, an image processing data path, 
a compression/decompression engine, a resize circuit, a 
display processing circuit, and a rotation circuit. Data from 

35 the CCD is routed to the capture data path for processing. 
The processed data is then sent over a main bus to be stored 
in an input buffer. The data flow control finds the appropriate 
image data for retrieval. Further processing is performed 
(e.g., decompressing, line averaging, pixel shuffling, ring 

40 insertion, interpolation, edge enhancement, gamma 
correction, and color space conversion). A JPEG 
compression/decompression engine compresses the result- 
ing image before it is stored as a file. The JPEG engine can 
subsequently decompress a file for display. The uncom- 

45 pressed file can first be resized to suit the desire of the user 
and/or rotated, depending on the current physical orientation 
of the digital camera and that of the image. 

By selectively activating and reusing certain parts of the 
hardware architecture and various data paths, at least four 

50 modes of operation can be supported: live view, capture, 
instant review, and play mode. In a live view mode of 
operation, the image from the capture data path is stored in 
the input buffer of the memory, retrieved and processed by 
the image processing path, buffered, resized if necessary, 

55 and displayed along with any appropriate graphics. In a 
capture mode of operation, the image from the capture data 
path is stored in the input buffer of the memory, retrieved 
and processed by the image processing path, buffered, 
compressed by the JPEG engine, and stored in a file format. 

60 And in an instant review mode of operation, the image from 
the capture data path is simultaneously resized, if necessary, 
stored in a display buffer, and displayed along with any 
appropriate graphics. Lastly, in a playback mode of 
operation, one or more requested image files are read, 

65 decompressed, resized if necessary, and displayed along 
with any appropriate graphics (e.g., padding or overlay 
bars). 



04/27/2004, east version: 1.4.1 



US 6,563,535 Bl 

3 4 

An orientation detector is used to detect the current apparatus, digital VCRs, digital camcorders and recorders, 

physical orientation of the display device so that the image etc.). That is, any image capture device which displays 

from a first draw buffer can be rotated by the rotation circuit, images, icons and/or other items, could incorporate the 

in accordance with the current orientation, and stored in a features described herein below and that device would be 

second draw buffer. The correct image is automatically 5 within the spirit and scope of the present invention. Thus, the 

displayed in all modes, regardless of the orientation of the present invention is not intended to be limited to the embodi- 

image or the physical orientation of the camera (both at the ment shown but 15 10 be accorded the widest scope consistent 

time the picture was taken and at the time the picture is being ™ th tne P™ciples and features described herein, 

rendered for display). FIG. 1 shows the block diagram of a digital camera upon 

By utilizing the optimized hardware design and data paths « which the present invention may be practiced. Digital cam- 

of the present invention, faster image processing can be era 100 * ^mpnsed of a lens assembly 102, filter 103 

achieved with a wide range of features, modes of operation, *™ito,*n*toe P rocess0 [ 105 > and d 'S ltal 

and enhancements. Furthermore, with the novel layout of the processor (DSP) 106. The image of an object or scene 101 

hardware design and data paths of the present invention, ■ Raptured by reflected light passing through lens assembly 

certain parts and paths can be used in different capacities and « 10 * The light is then filtered by a color filter array 103 (e.g., 

in different modes of operation. This reduces the total a ^yer array) which provides both luminance and color 

amount of hardware which is needed, which minimizes the ^formation pertaining to the image. The image is then 

overall cost with minimal or no impact on speed and co °^ d m 10 ac ^^ctncal signal by image sensor 1W (e.g 

functionality. a CCD devicc )- ^ raw ima g e data ( c -8-» 10 CCD format) 

20 k tnen routed through an analog signal processor (ASP) 105. 

BRIEF DESCRIPTION OF THE DRAWINGS ASP 105 is an ASIC chip which performs several analog as 

well as digital functions: analog signal processing, bad pixel 

The operation of this invention can be best visualized by replacement, pixel averaging, white balance gain control, 

reference to the drawings. and analog-to-digital (A/D) conversion. The raw CCD 

FIG. 1 shows the block diagram of a digital camera upon 2 5 image data is then passed on to the digital signal processor 

which the present invention may be practiced. (DSP) 106. The DSP ASIC chip 106 combines the following 

FIG. 2 A shows a more detailed block diagram of the DSP related functions: front-end pixel data path to a frame buffer, 

and memory. statistics generation, image processing, compression/ 

FIG. 2B shows an alternative embodiment of the detailed decompression live view generation, rotation, resize, video 

block diagram of the DSP and memory. 30 generation and timing generation The passed image 

„ f . . . _ j . c from DSP 106 is displayed onto a built-in LCD 107. LCD 

FIG. 3Ashows the data processing flow diagram for a live m caQ ^ ^ a viewfinder and as a display for caplured 

view mode of operation. ^ imagc signa , caQ ^ ^ omput from Dsp 1Q6 

FIG. 3B shows the data processing flow diagram for a m e ; ther an njsc or PAL format, 

capture and instant review mode of operation when the 35 A main bus 113 is used to carry data transmissions and/or 

camera is in a horizontal, unrelated position. contro , signals between the DSP 10 6 and motor controller 

FIG. 3C shows the data processing flow diagram for a 108, memory 109, central processing unit (CPU) 110, input/ 

capture and instant review mode of operation, whereby the output (I/O) device 111, and rotation sensor 112. The motor 

camera is in a rotated position (e.g., portrait state). controller 108 controls the motor used in digital camera for 

FIG. 3D shows the data processing flow diagram for the 40 physically adjusting the lens assembly 102. Memory 109 is 

post unrotation of an image for instant review. dynamic random-access memory (DRAM) and can include 

FIG. 3E shows the data processing flow diagram for the either non-volatile memory (e.g., flash, ROM, PROM, etc.) 

post rotation of an image for iastant review given that the and /° r removable memory (e.g., memory cards, disks, etc.). 

image is not found in the draw buffer. Memory 109 is used to store raw image digital data as well 

FIG. 3F shows the data processing flow diagram for a « " pressed image digital data. CPU 110 is a processor 

normal, unrotated play mode of operation. £ f> lhe u 823 processor manufactured by Motorola Inc. of 

n „ , , ' . . Scnaumberg, 111.) which can be programmed to perform 

FIG 3G shows the data processing flow diagram for a various ^ * MaodMeA with me di ilal camera 100 . ^ I/0 

rotated play mode of operation. deyice m as aQ mlerface> whereby the user> tDrougn 

FIG. 3H shows data path flow diagram for generating 5Q (ne ^ of bu(tons , menus> arrows, overlays, cursors, 

compressed screen nail or thumb-nail images. prompts, etc., can control various functions of the digital 

FIG. 31 shows data path flow diagram for generating camera 100. Rotation sensor 112 senses the current orien- 

thumb-nail from screen-nail. union of the digital camera 112 so that captured images can 

FIG. 4 is a detailed block diagram of the currently be automatically rotated to the desired landscape or portrait 

preferred embodiment of the digital signal processor. 55 format for display. It should be noted that there are many 

different configurations which can be used to practice the 

DETAILED DESCRIPTION present invention. In one embodiment, the CPU and the DSP 

An imaging system for high performance digital imaging 106 reside on a single chip. In other embodiments, the CPU 

devices is described. The following description is presented and DSP reside on two or more separate chips. What is of 

to enable one of ordinary skill in the art to make and use the 60 particular importance is the hardware architecture found in 

invention and is provided in the context of a patent appli- the DSP block 106. 

cation and its requirements. Although the present invention FIG. 2A shows a detailed block diagram of the preferred 

is described in the context of a digital camera, various embodiment of the DSP 106 and memory 109. The capture 

modifications to the preferred embodiment will be readily data path block 201 basically interfaces with the image 

apparent to those skilled in the art, and the generic principles 65 capture head (e.g., the CCD). Some of its functions include 

herein may be applied to other embodiments (e.g., digital controlling the CCD driver, performing slight compression, 

scanners, video teleconferencing circuitry, multimedia collecting statistics regarding the raw CCD data, generating 
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timing references, aod loading the input buffer 210 with the and versatility such that it can support all four modes of 

slightly compressed raw data. An optional second input operation: live view, capture, instant review, and playback, 

buffer 211 is provided to increase performance using stan- The live view mode causes the actual image that is seen by 

dard "Ping-Pong" buffering techniques. When the image is the camera in real time to be displayed on the camera's LCD. 

subsequently requested, it is the job of the data flow control 5 The capture mode of operation stores a picture when the user 

block 203 to locate and fetch the appropriate CCD image clicks the shutter button. The instant review mode of opera - 

data from the input buffers 210 or 211 and pass this raw data tion displays an image immediately after the shutter button 

to the image processing data path 202. It is the function of is clicked by the person taking the picture. The user can 

the data flow control block 203 to transform raw lines into immediately review the picture that was just taken. If the 

blocks of pixels called minimum computational units user is not satisfied with the picture, he/she may simply 

(MCU). The imaging data path block 202 basically takes the discard that picture and shoot again. Also, capture instant 

raw CCD data and converts it into a real image data capable review are essentially simultaneous with this architecture, 

of being displayed. Some functions involved in this conver- allowing it to be used to "show" a burst real time as it is 

sion process include edge enhancement, pixel interpolation, captured. The play back mode sequentially displays a series 

ring insert, pixel shuffle, line averaging, gamma correction, of stored pictures on the LCD. For example, a user may take 

and color space conversion. The final output is displayable 15 a rapid series of shots of some event (e.g., a golf swing) and 

image data (e.g., in YCC format). later playback those time-lapsed images. The captured 

A buffer 204 is used to coordinate the transfer of display- images may be captured in full, quarter, or one-sixteenth 

able image data to the JPEG block 205, which compresses size. 

the data. The compressed displayable image data is then Another advantage offered by the present invention is that 

stored in file buffer 213. The displayable image data can 20 the present invention provides automatic processing and 

subsequently be retrieved from file buffer 213 and decom- correct display of the image, regardless of the orientation of 

pressed by JPEG block 205. If necessary, the displayable the camera (e.g., portrait, landscape, or upside-down) at the 

data can be resized (e.g., enlarged or reduced) by resize time an image was taken and also regardless of the orien- 

block 206. Display processing 207 then performs the req- tation of the camera at the time an image is subsequently 

uisite graphic functions on the image data in order to 2 5 rendered for display. The present invention handles live 

generate the final LCD display (e.g., the preferred embodi- view generation with the camera in either portrait (vertical) 

ment is to prepare image data for graphical overlay and do or landscape (horizontal) orientation; image capture with 

actual graphics by software during the final rendering step, camera rotation or without camera rotation; instant review 

padding, overlays, menus, prompts, date and time stamp, with image rotation, without image rotation, with camera 

etc.). Note that the graphics overlay and layouts can also be 30 rotation, or without camera rotation; and play mode with 

rotated and resized accordingly to fit onto the screen. The image rotation, without image rotation, with camera 

LCD display data is temporarily stored in draw buffer 214 rotation, or without camera rotation. According to the 

before being rendered for display on the LCD screen. If the present invention, all of the above modes and orientations 

image needs to be rotated (to the right or left 90 degrees) to can be accomplished through the use of the same hardware 

obtain the correct landscape or portrait view, the displayable 35 architecture. Each of these four modes of operation, their 

image data is retrieved from the draw buffer 214, and its respective data processing flows, and requisite active blocks 

pixels are swapped accordingly by the YCC shuffle block are discussed in detail below and in reference to FIGS. 

208. The rotated image is then temporarily stored in the 3A-3F and FIG. 4. 

rotate draw buffer 215. It should be noted that some of the Not only does the present invention provide for all the 

blocks 201-208 can be implemented in software. For 40 various modes, orientations, and display sizes, but it also 

example, a portion of the display processing can be per- leverages each of the blocks 202-208 such that they can be 

formed in software. Although the goal is to increase the used interchangeably from one mode/orientation/size to the 

overall speed, a part of this architecture can, nevertheless, be next. Depending on which particular mode/orientation/size 

implemented as a software process for simplification, size is currently desired, certain ones of these blocks 202-208 are 

reduction, logic minimization and flexibility purposes. Gen- 45 activated to achieve the desired results required for that 

erally the most computationally complex blocks which are particular mode/orientation/size. In other words, one can 

least likely to be replaced by software include the JPEG 205, selectively utilize certain combinations and data flows to 

image processing data path 203, and resizer 206 blocks. achieve a wide range of desired effects. Hence, the total 

Conversely, the entire or parts of the data flow control 203, amount of hardware that is needed to provide all of the 

display processing 207, and/or YCC shuffle 208 blocks can 50 above functionality's can be achieved efficiently and eco- 

be implemented as a software process. nomically with minimal impact to bus bandwidth and the 

There are several advantages which makes the hardware camera's overall processing speed. As an example, the resize 

architecture of the present invention superior to other con- block 206 is optimized to perform resizing functions. As 

figurations. Namely, the same path is used to process live such, it is used in live view, instant review (rotated and 

view as well as to process subsequent capture, instant 55 unrotated), and play (rotated and unrotated) modes, 

review, and playback views. Bus 113 is used to convey However, it is not used during the capture mode. Likewise, 

incoming real-time data used for a live view display as well the YCC shuffle block 208 is used during the instant review 

as data that had been captured and stored as a file. Thereby, (rotated) and play (rotated) modes. The JPEG block 205 is 

combinations of blocks 202-208 coupled to bus 113 can used to compress data from the MCU buffer for storage in 

accept and process both real-time CCD as well as previously 60 the file buffer 213 during the capture and instant review 

captured and stored file data. Because of this lack of (unrotated and rotated) modes. Additionally, JPEG block 

differentiation, the display of real-time images is exactly the 205 is used to decompress data retrieved from file buffer 213 

same as the display of images for instant review, capture, during the play (unrotated and rotated) mode. A detailed 

and playback modes. There are no annoying discrepancies description describing how, when, and why each of the 

between the various displays. 65 blocks 202-208 arc used follows. 

Another primary advantage is the fact this one hardware FIG. 2B shows an alternative embodiment of the DSP and 

architecture of the present invention has enough flexibility memory, whereby a direct path to the MCU is provided. 
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Referring now to FIG. 3 A, ihe data processing flow rently stored in the draw buffer 214. The image may have 

diagram for a live view mode of operation is shown. The live been stored in draw buffer 214 as part of the processing flow 

view mode displays on the LCD screen the image that is associated with the capture mode described above and in 

seen by the camera in real time. As the camera is pointed and reference to FIG. 3C. If the correctly rotated image without 

moved, the scene on the LCD display changes accordingly. 5 graphics, is found in draw buffer 214, it is retrieved and 

The raw CCD data from the image capture head is input to passed through block 208 with YCC shuffle turned off. This 

the capture data path 201, processed, and stored in the input image is then fed into MCU buffer 204 and then resized 3:4 

buffer 210. The data flow control 203 retrieves this data by resizer block 206 to fit it onto the screen. Display 

which is processed by the image processing data path 202 processing block 207 generates the appropriate graphics 

and sent to the MCU buffer 204. A resize block 206 is used 10 overlay which is stored along with the image in rotated draw 

to fit the image captured by the CCD onto the LCD's image buffer 215. Eventually, this data is read out for display, 

size. The requisite display processing is performed in block FIG. 3E shows the data processing flow diagram for the 

207. In the preferred embodiment, when the camera is held post rotation of an image for instant review given that the 
horizontally (i.e., landscape), the image is temporarily stored image is not found in the draw buffer or it only exists with 
in the draw buffer and fit immediately rendered out for 15 graphics data. Assume as with the previous case that the 
display on the LCD. In a vertical orientation, the data flow camera is in a rotated orientation when the shutter was 
control block 203, during live review, ignores the state of the clicked to capture an image. Now, user rotates the camera 
orientation sensor. Thus, the image flow is straight forward back to an unrotated orientation (e.g., landscape) for instant 
because the simultaneous rotation of the CCD and LCD review of the captured image. Assume also that the image is 
display obviates the need to perform image rotation by the 20 not found in the draw buffer (e.g., the image in the draw 
hardware. In one embodiment, the data flow control block buffer was discarded in order to free up additional memory). 
203 determines the order by which data is fed into the image In this case, the image in the form of a JPEG file 213 is 
processing data path 202, thereby effecting the rotation of processed by JPEG engine 205 before being fed into the 
the image. MCU buffer 204. Resizer 206 is used to fit the image, which 

FIG. 3B shows the data processing flow diagram for a 25 is rotated with respect to how it was captured (e.g., portrait), 

capture mode of operation when the camera is in an to the correct size of the screen. In this first pass, the display 

unrotated, horizontal orientation. The capture mode of processing 207 is turned off so that graphics is not generated, 

operation stores a picture when the user clicks the shutter The image is then temporarily stored in the draw buffer 214. 

button during the live view mode of operation. The block In a second pass, represented by the dashed lines, the rotated 

diagram for capture mode is basically the same as for the 30 image in the draw buffer 214 is fed through block 208 with 

live view mode described above, except that the image data YCC shuffle turned off. After passing through MCU buffer 

from MCU buffer 204 is compressed by JPEG block 205 and 204, the image is reduced by 35% so that it will fit into the 

stored in JPEG file buffer 213. In addition, the image data landscape mode screen. Otherwise, it would be too tall to fit. 

from MCU buffer 204 is resized (if necessary) by block 206; The appropriate display processing 207 is performed to 

graphical information is added by display processing block 35 generate the graphics. Thereupon, the graphics and pro- 

207; and the final image is stored in draw buffer 214. cessed image is stored in the rotated draw buffer 215 before 

FIG. 3C shows the data processing flow diagram for a being drawn for display. Note that both passes can be 

capture mode of operation when the rotated camera is in a accomplished in one step if the needed resize ratio is 

vertical orientation when the shutter button is pressed. In this available. 

case, a second pass is needed. A first pass is performed in the 40 FIG. 3F shows the data processing flow diagram for a play 

same manner as described above for the unrotated capture mode of operation when the image and camera are both in 

mode of operation except that data flow control block 203 horizontal orientation or both are in vertical orientation. In 

rotates the image from the frame buffer (always landscape) a play mode, one or more pictures are sequentially rendered 

to the current rotation of the camera (left or right portrait or for display on the LCD. The requested picture or pictures are 

upside down). Also, the display processing block 207 post- 45 retrieved from the JPEG file buffer 213 and decompressed 

pones generation of it's graphical information. A second by JPEG block 205. The decompressed image data is then 

pass is then performed after the first pass. The second pass sent to the MCU buffer 204 which coordinates the data 

is indicated by the dotted lines. In general, the image data transfer to the resize block 206. The images can then be 

from the draw buffer 214 is unrotated by YCC shuffle block scaled down (e.g., to a LCD size) or zoomed (i.e., enlarged) 

208, which is turned on. The resulting image data is stored 50 by resize block 206. The requisite display processing by 
back to the MCU buffer 204. From the MCU buffer 204, the block 207 is performed, and the image is then sent to the 
image data is fed into resize block 206. On the second pass, draw buffer 214 to be rendered for display on the LCD 
a 1:1 resize is performed since the resizing was already screen. 

performed during the first pass. In effect, the rotation for FIG. 3G shows the data processing flow diagram for a 

instant review is canceled because the camera is also rotated. 55 play mode of operation when the camera is currently in a 

During the second pass, the display processing block 207 vertical orientation and the image is in a horizontal orien- 

generates the graphical information. Thereupon, the image tation or if the camera is in a horizontal orientation and the 

and graphical data is stored in the rotate draw buffer 215 and image is in a vertical orientation, as determined by the 

subsequently rendered out for display. rotation sensor. In either case, the same flow is followed as 

FIG. 3D shows the data processing flow diagram for the 60 described above in reference to FIG. 3E, except that a 

post rotation of an image for instant review. If the camera is second pass is performed and display processing if turned off 

rotated back to normal (landscape) orientation while the on the first pass. For the sake of clarity, only the flow 

instant review is still active, (i.e., the camera has not diagram corresponding to the second pass is shown. The 

returned to live view mode) the CPU can be programmed to image data in draw buffer 214 is fed back to the YCC shuffle 

delect this by polling the orientation sensor. If this state is 65 block 208. The YCC shuffle block 208 rotates the image by 

delected, a rotated image can be generated, as described shuffling the pixels. The rotated image is then resized by 

below. The software determines whether the image is cur- resizing block 206 because its dimensions have changed. 
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After the appropriate display processing by block 207, the 
rotated, resized image is then sent to the rotate draw buffer 
215 to be rendered for display on the LCD screen. 

FIG. 3H shows a data flow diagram for generating com- 
pressed screen-nail or thumb-nail images. There is a direct 
path from the YCC block 208 to the MCU block 204. This 
allows resized images to be compressed (e.g. screen-nails). 
Note that this only allows compression or resizing of exist- 
ing image; rotation or unrotation would be performed in 
software. Alternatively this path may be eliminated by 
performing the same function in software. 

FIG. 31 shows a data flow diagram for generating a 
thumb-nail image from a screen-nail image. 

It can be seen based on FIGS. 3A-3I and the descriptions 
given above that the same set of blocks 201-208 or circuits 
can be used to achieve virtually any mode, orientation, or 
display size for a digital camera. The only thing that changes 
is the selection of the appropriate data path(s) and the 
activation of selected blocks. Although the Goal hardware 
architecture of the present invention appears straightforward 
in retrospect, it was deceptively difficult to conceive. This is 
because one has to first decide on which blocks are to be 
implemented and define each block's respective functions. 
Duplication must be minimized. Next, the data paths must 
be laid out so as to interconnect the various blocks in such 
a manner that certain combinations produce all of the 
desired results correctly. Flexibility, adaptability, and high 
functionality are key. Moreover, the data must be routed 
through the various data paths as fast as possible so that 
there is no bottleneck in any one path that might reduce the 
overall bandwidth. Other considerations include which ones 
of the blocks should be activated and under what circum- 
stances should they be activated. All these factors were taken 
into account in order to produce the integrated solution of 
the present invention which effectively and efficiently uti- 
lizes the available hardware resources. 

The actual circuits that were implemented for each of the 
blocks are now described with reference to FIG. 4. It should 
be noted, however, that there exist many different circuits 
which can be used to implement the present invention. For 
example, there are many different ways in which a resize 
circuit or a rotate circuit can be designed. The following 
description relates to the currently preferred embodiment of 
the digital signal processor 106. In general, the capture data 
path comprises of the FIFO 401, AE/AWB statistics 402, 
pixel average 403, 10 to 8 LUT 404, and 16x32 FIFO 405 
blocks. The image processing data path and data flow 
control consists of the edge enhance 407, 5x5 interpolate 
408, 1320 pixel MCU buffer 409, ring insert 410, pixel 
shuffle 411, line average 412, 8 to 10 look-up-table (LUT) 
413, CCIR 709 gamma, and color space converter 415 
blocks. 

In the currently preferred embodiment, data is processed 
in blocks known as MCU's-minimum computation units. An 
MCU is defined and well known in the art of image 
processing, especially as they relate to YCC 411, YCC 422, 
and discrete cosine transformations (DCTs). The primary 
method of achieving processing compatibility between these 
data paths involves storing raw CCD data from the capture 
data path into memory in line format to support image 
rotation while reducing bus bandwidth, and to then trans- 
form the raw lines in memory into raw MCU's, which are 
necessary to support the image processing and compression 
chain. Each of the specific blocks are described in detail 
below. 

Starting with the raw CCD data generated by the CCD 
array (or some other equivalent image capture head), this 
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data is input to the pixel average block 403. The pixel 
average block 403 performs pixel averaging to reduce lines 
to half and quarter length. The averaging is performed by 
adding values and shifting as follows: 

Full Output: Ra, Ga, Rb, Gb, Rc, Gc, Rd, Gd 
Half Output: (Ra+Rb)/2, (Ga+Gb)/2, (Rc+Rd)/2, (Gc+ 
Gd)/2 

Quarter Output: (Ra+Rb+Rc+Rd)/4, (Ga+Gb+Gc+Gd)/4 
It should be noted that this block 403 does averaging of 
pixels on a line. The compliment function, line averaging, is 
done when the data is returned to the ASIC for image 
processing. In this case, MCU's are returned, making it easy 
to do the line averaging. This dual process eliminates 
aliasing in the image for 1/4 and 1/16 resolution captures. 
For 1716 capture, CCD line skipping (e.g., 2 out of 4) may 
be used without serious image quality effects. However, for 
1/4 capture, line skipping should not be used. During live 
view mode, typically the CCD is being scanned using line 
skipping, for 50% vertical resolution, reduced bus 
bandwidth, and faster frame rate. As can be seen from Table 
1 below, for CCD's 1 152 and larger, Vi6 mode is used for live 
view. To accomplish this, a combination of 2 out of 8 line 
skipping and Quarter Output is used. This is necessary to 
provide sufficient read-out speed for large CCD's, as well as 
keeping the bus bandwidth low enough during live view. It 
should be noted that using 1/4 or 1/16 size in live view is 
optional. 

TABLE 1 

CCD Size and Live View Size Table 
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45 



50 



60 



65 



Image Size 



V* 

Image Size 



l A6 

Image Size 



M 
Pixels 



640 


480 


32Q 


240 


160 


120 


031 


768 


576 


384 


28£ 


192 


144 


0.45 


896 


672 


448 


m 


224 


168 


0.61 


1024 


768 


512 




256 


192 


0.79 


1152 


864 


576 


432 


2fiS 


m 


1.00 


1280 


960 


640 


480 


i2Q 


240 


1.23 


1536 


1024 


768 


512 


m 




1.58 


1536 


1152 


768 


576 




2sa 


1.77 


1792 


1344 


896 


672 


448 


m 


2.41 


2048 


1536 


1024 


768 


512 


384 


3.15 



•Live view sizes to resize r are shown underlined 

The AE/AWB statistics block 402 receives its data from 
the Pixel Average block 403 (either the Half Output or 
Quarter Output, depending on CCD size). This size is used 
by the statistics circuit. This is done by selecting one of two 
outputs from the Pixel Average block 403, and doing either 
2/4 or 2/8 line skipping. Since actual line skipping may also 
be done, this setting must be set taking into account the 
Timing Generator 406 setting. In any case, the resulting data 
used by the statistics circuit is between 288 wide and 512 
wide, with the corresponding height. The AE/AWB block 
402 generates an 8x6 set of averages for normal 4:3 CCD's, 
and 8x4 set of averages for 3:2 CCD's. This is accomplished 
by varying the block size depending on the CCD size. 
Furthermore, the AE/AWB Statistics block 402 needs to be 
capable of handling the pixel offset during a rotated capture 
sequence. In addition, it optionally collects "sum of absolute 
differences" for each of the statistics blocks to assist in 
providing automatic focus. 

The FIFO 8x32 block 401 receives the data from the 
AE/AWB statistics block 402 and requests a DMA transfer 
when the buffer has 32 bytes (half full). This FIFO 401 
supports quad word transfers. The pixel average block 403 
also supplies its 10-bit data to the 10 to 8 LUT block 404, 
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which converts it into an 8 -bit compressed data. This LUT For example, if the CCD is 1152x864, the steps supported 

block 404 reduces the memory footprint for CCD data, but are: 1024, 896, 768, 640, 576 (2x), 512, 448, 384, 320, and 

without loss of image quality. 288 (4x). Furthermore, the timing generator 406 has many 

Thereupon, the 16x32 FIFO 405 takes data from the LUT control registers, which must be accessible via the system 

404 and requests a quad or octal DMA transfer when half s bus or a second control bus. 

full. In addition to the octal transfer, the DMA for this data Data from memory is fed via the BIU 422 to the 8 to 10 

also supports interlacing. This is required since many large LUT block 413, which takes data from the frame buffer and 

CCD's utilize an interline scan method for full resolution decompresses it back to 10 bit linear data. This may require 

capture, where 25% or 50% of the lines are transferred per a RAM table look-up. Alternatively, a specific version can be 

scan. 10 produced to reduce ASIC size. 

The data from FIFO 405 is then sent to the memory over From the LUT block 413, the data is sent to the ring insert 

a 32-bit bus via the bus interface unit (BIU) 422. Most 410, pixel shuffle 411, and line average 412 blocks. It should 

transfers over this bus between memory and the DSP are in be stressed that there are many different ways, besides these 

quad, octal, or programmable length bursts to increase bus three particular blocks, in which to implement these func- 

bandwidth. In addition to the DMA interface, there are a is tions. First, the line average block 412 averages 4 lines into 

number of registers inside the DSP ASIC which are acces- 2, or simply passes the data through without change. When 

sible by software. The BIU 422 gives bus read/write access this function is turned on, the DMA feeds twice as many 

to these registers. See U.S. application Ser. No. 08/916,186, lines per block. The number of lines of pixels required is 

"Method & System for Organizing DMA Transfers to Sup- defined in the ring pixel insert block 410. This block is used 

port Image Rotation", filed Aug. 21, 1997, assigned to 20 in conjunction with timing generator line skip 2/4 or 2/2 and 

assignee of the present application and incorporated herein pixel average 2/4 or 2/8 to build high quality 1/4 and 1/16 

by reference. In another embodiment, access can be size images for capture. Normally, for live view, this func- 

achieved through a serial port. In this case, double buffering tion is turned on to produce an image size for 288 to 512 

of registers is required, due to slow transfer rates. Transfer horizontally. The mode selected depends on the CCD size, 

between buffers is activated at frame boundaries. 25 Of course, one can implement a resizer capable of resizing 

The liming generator 406 supports 2/4 and 2/8 line to any given resolution, 

skipping — either by actual CCD line skipping or by not When the line average function is turned on, the DMA 

clocking data from the A/D into the DSP ASIC, depending moves blocks which are twice as high (i.e., contains twice as 

on CCD capability. In addition, the active pixel election many lines) as when the line average function is turned off. 

during capture takes into account the required offset for the 30 These lines are averaged down, using a similar process as 

Bayer pattern for left or right portrait rotation. This requires used for pixel averaging. This is demonstrated for two pixels 

shifting the active area either one column right or one line per line below: 
down, depending on which rotation is required. 

FIG. 5 shows how the active pixels are selected for a 8x6 

imager, assuming the landscape orientation. As shown, the 35 
pixels used are exactly 8x6. Ring pixel insertion is done at 
a different stage in the process. The extra "available" pixel 
and line are unused in this case. A left portrait is when the 
user rotates the camera clockwise, looking from the rear. 

This requires the lop of the portrait image to be on the left 40 
side of the landscape image. The image data shuffle is a right 

rotate in this case. A right portrait is when the user rotates the The line average block 412 requires a single control bit: 

camera counter-clockwise, looking from the rear. This l=line average, 0=off. 

requires the top of the portrait image to be on the right side Second, the pixel shuffle block 411 rearranges the data for 

of the landscape image. The image data shuffle is a left 45 portrait left and right cases and in one embodiment, upside 

rotate. down case as well. This is illustrated for 4x4 blocks, as 

FIG. 6 shows how the pixel selection is shifted for left and shown in FIG. 8. This process can be accomplished by 

right portrait capture. The capture is shifted right one pixel addressing the data correctly as it is moved into the pixel 

for left portrait, and is shifted down one pixel for right MCU buffer 409. The pixel shuffle block 411 requires two 

portrait. This preserves the sequence of GB/RG once the 50 control bits to indicate rotation: (XMandscape, 10=left 

pixels are shuffled into the correct positions. In addition to rotate, 01=right rotate, and optionally ll=upside-down. 

the above modes, the "upside down" mode may also be Third, the ring insert block 410 is responsible for insertion 

supported. This mode is required for cameras where the lens of ring pixels at the edges of the image. The proximity to the 

can be rotated 180 degrees for a self-portrait. A sensor edge is provided by data in control registers. This informa- 

determines when the lens has reached 180 degree position, 55 tion is also required for the DMA function, since a different 

and tells the software to invert the image. Actually, the amount of data must be transferred when at the image edge, 

software "flips" the image both left to right and top to FIG. 9 shows how data is transferred by the DMA to support 

bottom. The pixel offset required for this is shown in FIG. data in Live View and Landscape Capture. Note that the first 

7. The data is actually upside down in the frame buffer. This and last pass across the data is different than all passes away 

can be corrected by either the DMA on the way into the 60 from the top and bottom of the image. This is because there 

frame buffer, or the DMA on the way out of the frame buffer. are no ring pixels in the frame buffer. Also note that each 

The Timing Generator 406 must also be able to set the successive pass rescans four lines from the previous row. 

active area independently of the actual CCD imager size. This is necessary to provide the full overlap of data for the 

This functionality is required for digital zoom. The present interpolator. The DMA must back up four lines at the 

invention allows all of the standard image sizes to be 65 beginning of each row to accomplish this. However, if the 

captured from a CCD which are equal or smaller than the line average function is turned on, the DMA will actually 

CCD size. This enables a 5-step zoom for each 2x of zoom. have read 36 or 40 lines of data, rather than 18 or 20. 
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A portrait DMA scan is similar to landscape, but proceeds 
in different directions. Since DMA cannot really transfer 
words on a non-longword boundary, it is assumed that all 
transfers will actually be 20 bytes (burst length of 5). 
Alternatively, if two quad transfers are required because of 5 
system limitations, all additional data must be discarded. 
This method increases bus loading significantly and is thus, 
not the preferred method. For the first and last "row", the 
extra 2 bytes will be discarded. The blocks are the same size 
for portrait as landscape once they reach the MCU pixel 10 
buffer 409, and ring pixel insertion is done in the same way. 
However, the data is read in by columns rather than rows. 
The data is rearranged in the Pixel Shuffle block 411. The 
DMA transfers 32 lines of 20 bytes each for portrait mode. 
But support for fewer than 32 lines is required for the end of 15 
a scan row. In which case, there is available a fiill set of 
eighteen pixels at the last "row", since all image widths 
supported are divisible by sixteen. Pixels outside of the 
MCU at the right or along the bottom, assuming that the 
image is not a multiple of 16, are filled with black or are 20 
appropriately handled in subsequent blocks. 

Next, the pixel MCU buffer 409 holds the data for 
processing by the interpolate block 408. The 5x5 interpolate 
block 408 performs a 5x5 matrix operation from 20x20 
inputs to form a 16x16 MCU of RGB data. The exact form 25 
of the matrix values must be determined. The number of bits 
of resolution for the matrix coefficients must also be 
determined, given a 10 bit input resolution. The preferred 
embodiment includes having the interpolation block 408 and 
edge enhancement block 407 be a programmable DSP, 30 
thereby allowing various algorithms to be used. During the 
interpolation process, all results must be clipped to the 
maximum and minimum values. 

Block 407 performs edge enhancement. This effectively 
detects edges in different directions when computing aver- 35 
ages for missing color restoration during the interpolation 
process. This process is best integrated into the Interpolation 
process of block 408. The output from these blocks 407-408 
is preferably CCIR 709 RGB data, having been recon- 
structed from the Bayer pattern, color corrected, feature 40 
enhanced, and gamma corrected. Fine AWB can also be 
performed in this stage, if required, by adjusting the matrix 
values appropriately. There is one R, G, and B output value 
for each of 16x16 pixels. This data is transferred directly to 
the gamma correction block 414 as it is computed, pixel by 45 
pixel. 

It is the function of the color space convert block 415 to 
convert the 8-bit gamma corrected data from RGB space to 
YCC 422 space (subsampling to 411, when selected, is done 
as part of the JPEG block 205). The output from this stage 50 
is a 16x16 array of Y values, and two 16x8 arrays of Cr and 
Cb values. This gamma correction block 414, in its simplest 
embodiment, is essentially a 3x3 matrix with programmable 
matrix values. Assuming that the JPEG compression engine 
205 processes one block while the other block is being 55 
loaded, double buffering is performed. Thus, two 512 byte 
buffers are used. The registers for this block 414 are nine 
coefficient registers. Assuming 10 bit values, these can be 
arranged as three 30-bit registers. Note that different color 
space conversions can be supported: Photo YCC for 60 
FlashPix, and CCIR 709 for JPEG. Both of these transfor- 
mations are required at the same time when Photo YCC is 
used, since the data going to the resizer for Live View or 
Instant Review must always be CCIR 709. In addition, 
during decompression (playback support), Photo YCC must 65 
be converted to CCIR 709, since data space used in the 
frame buffer is only CCIR 709. A determination must be 
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made if the same block with programmable coefficients can 
do both conversions. It may be necessary to have three 
blocks: RGB to Photo YCC, RGB to CCIR YCC, and Photo 
YCC to CCIR YCC. Alternative embodiments of the image 
processing stages from the MCU input buffers to the YCC 
buffers are contemplated to be within the scope of the 
present invention. Furthermore, different solutions are 
possible, depending on the required quality of the conver- 
sion. 

The YCC MCU buffer 204 holds one or two MCU's in 
YCC 422 format. In order to support custom processing of 
images as they pass from conversion to YCC on to 
compression, direct access to this memory is preferred from 
the CPU. In this case, the CPU must be able to tell the JPEG 
205 and resizer 206 blocks when the custom modifications 
are completed on a block. Also, an interrupt must be used to 
tell the CPU when a block is available for processing. It may 
be preferable to require custom processing on uncompressed 
data in memory. This requires a buffer twice the size of the 
capture buffer, however, but allows full image access for 
simplified image processing rather than MCU access. 

The JPEG/IJPEG block 205 performs either a JPEG 
compression of an MCU, or a JPEG decompression of an 
MCU. In the first case, data is retrieved from the MCU 
buffer 204, and the results are output to the FIFO. In the 
IJPEG case, the data is taken from FIFO 416, and output to 
the MCU 204. The JPEG/UPEG function can operate in 422 
and 411/420 modes via a programmable register. When 
operating in 411/420, the JPEG block 205 must do the 
averaging from 422 to 411, or IJPEG up sample from 411 to 
422 in the preferred embodiment. The MCU block 204 is 
only a 422 format block. A control register must be included 
to set for 411 compression vs. 422. Additionally, two control 
bits must be included to set the data flow direction (i.e., 
compress, decompress, or off). 

The resize block 206 takes as input the MCU data from 
either the color space convert 415, IJPEG, or YCC shuffle 
208 blocks, and resizes and repackages it for use in the LCD 
frame buffer. The LCD buffer may be operating in 216 line 
1/2 vertical resolution mode (single repeated field) or in 432 
line full interlace mode. Both modes may be supported by 
this block. In either case, the horizontal resolution is 576 
(YCC 422). The buffer is thus 1152 bytes by 216 lines or 
1152 by 432 lines. This covers the "safe area" of NTSC 
video. Note that this data must always be CCIR YCC. Thus, 
for playback mode, where Photo YCC is being generated by 
IJPEG, conversion back to CCIR YCC must occur before the 
data goes to the Resizer 206. 

There are four resize cases to consider: 

1) Live View mode: data is always in landscape mode, the 
input data is always from 288 to 512 pixels wide, and 
the LCD buffer is always in 1/2 vertical resolution 
mode (216 lines). 

2) Capture mode and Instant Review: data can be in any 
of the three orientations, but must still be viewed in 
landscape mode, since this is still Live View of the data 
as it is being captured. Therefore, this is also always in 
1/2 vertical resolution mode (216 lines). Since the 
image processing section of the ASIC may have rotated 
the image for processing, the data must be "unrotated" 
for display on the LCD. This is done as a second step, 
using the memory to memory rotation path to be 
described below. 

3) Playback mode: data can be in any of the three 
orientations, and must be resized depending on orien- 
tation. For Landscape, the image should be 1152 bytes 
or 576 pixels wide (YCC 422) by 432 lines high. For 
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Portraits, the image should be 648 bytes or 324 pixels for PAL mode. This is accomplished using a line buffer 
wide by 432 lines high. Note that half of the lines are (1280 or 1152 bytes) and stopping the 8xx video controller 
tossed (not output to memory) if 1/2 vertical resolution during the sixth line. The currently preferred method is 
mode is being used. bilinear interpolation. 
4) Playback zoom mode: this is the same as Playback s .The LCD/Video generator Wock 417 generates composite 
ov «, nl ,u ; m , no j; rn i„ lllf i ,- c , mm „ui P nLmu video, component video, and RGB data for direct drive 
mode, except the image displayed is a movable wmdow ^ fio £ ^digital forms are available. In an 
on an image. Hi is is either the 1 to 1 mode where 1 alternalive embodiment, an "anti-flicker" block can be 
LCD pixel equals 1 image pixel, or the 1 to 2 mode, added to lhis fu nclion . Another alternative feature for the 
where 4 LCD pixels equals 1 image pixel. If half LCD/Video generator is fade capability. This can be a 
vertical resolution (216 line) is used, half of the lines io multiplier on the Y channel. The Y value would be multi- 
will be thrown away. plied by a value from 0 to 64 for a 64 step fade. A 256-step 
A 2K buffer 419 is interposed between the resize block multiplier value could be supported, so the fader software 
206 and padding overlay bars block 420 to temporarily store could use the proper "curve" to effect a linear visual effect, 
data. Since less than 1152 bytes per line may be generated This function could be used for fade-in and fade-out capa- 
by the resizer 206 in some cases, this block must pad the left 3S bility during playback. In another embodiment, a separate 
and right side of the data to fill the LCD. The assumption is data path be added to process graphic overlay buffers and 
that any padding on the top and bottom will be handled by 1152x216 image buffers into a single 576x216 data format, 
software by setting the appropriate starting address. Further, This would reduce the bus bandwidth by 10% during LCD 
it is assumed that hardware padding will only be required refresh. 

when the starting address cannot be used to center the image. Hence, an image processing system for high performance 

This is only true for 16 byte padding. This is the case for digital imaging has been described. The foregoing descrip- 

320, 448, 640, 896, 1280, and 1792. This also assumes that tions of specific embodiments of the present invention have 

octal DMA is used and must be on 32 byte boundaries. If been presented for purposes of illustration and description, 

quad DMA is used with 16 byte boundaries, hardware They are not intended to be exhaustive or to limit the 

padding will not be required. invention to the precise forms disclosed, and obviously 

Overlay bar generation is another function of block 420. 25 many modifications and variations are possible in light of 

This should only be active when an entire LCD buffer is the above teaching. The embodiments were chosen and 

being generated. Thus, this function should be turned off described in order to best explain the principles of the 

when images smaller than 288x216 are being resized. The invention and its practical application, to thereby enable 

overlay function is capable of producing two "bars" of others skilled in the art to best utilize the invention and 

image data at 50% luminance (Y value divided by 2). This 30 various embodiments with various modifications as are 

is defined by two "start fine" and "end line" values. suited to the particular use contemplated. It is intended that 

A data path is shown in the block diagram for data to the scope of the invention be defined by the Claims 

DMA directly into the YCC Shuffle block 208, and then into appended hereto and their equivalents, 

the MCU buffer 204. This path allows in-memory image What is claimed is: 

data to be rotated and resized, or compressed. The primary 35 1. A method for processing an image in an image capture 

purpose for this path is to rotate the LCD buffer image unit including a display, the method comprising the steps of: 

generated during portrait capture. This case is the "instant capturing raw image data; 

review" while the camera is still being held in the portrait processing the raw image data through a capture data 

position. If the camera is then turned into the landscape h whefcin lhc mre dala h stalislic 

position, the : image : must resized and rotated Tlie resize gene ration, pixel averaging, and compression; 

factor is 12/16 (3/4 ratio). It can also be used to generate & . *\ . . c 4 . . 

compressed screen-nail, if desired. sl0 ™$ lhe raw "nage dala in a first memory, wherein the 

The pixel shuffle block 208 performs a different kind of first raemorv conlams raw ima S e data for one or more 

shuffle here than for the Bayer pattern normally transformed. images; 

Basically, the YCC 422 data arc broken up into a 16x16 accessing a particular raw image data from the first 
block of Y's, and two 8x16 blocks of Cr and Cb. These are *5 memory and performing image processing on the par- 
shuffled. The Cr/Cb blocks are transformed after shuffle licular raw image data to produce a rotated processed 
from 16x8 to 8x16. An averaging or linear interpolating image; 

filter may be used for this transformation. Finally, the YCC loading the processed image in a second buffer; 
Shuffle block 208 should have a bypass mode, allowing the compressing the processed image to generate a corn- 
data path to be used for resizing or compression only. This 50 pressed image* 
would be used to do second -pass resize for already-rotated * . ' 
JPEG images. Since the initial resize will always be to the unrotating tne image; 

LCD buffer size assuming landscape, a second pass through performing display processing to produce graphics for 

the resizer is required to reduce to 75% for portrait. This display; 

second pass is required assuming integer resize only in 55 storing the graphics and image in a fourth buffer; and 

Resize block (N out of 16). There are cases where thumbnail rendering the graphics and image from the fourth buffer 

generation would be desirable. One way to accomplish this f or display on the image capture unit, 

is to run an image through the resizer 206 a second time. For 2. A method for processing an image in an image capture 

example, to generate a 3x3 thumbnail display of a "event umt including a display, the method comprising the steps of: 

bracket" capture, resizing 16:5 would give a good result. An capturing raw image dala; 

alternative embodiment is to have a direct path only, or YCC 60 ^„ mtfC : nn t u« • _ Ant ^ .Um,~u * ™,™ 

t_m / c >n • » j « rj. processing the raw image data through a capture data 

shuffle (rotate) is done in software. This is a good trade-off th- 

since the image is fairly small and can be processed io Pf ■ m . , ... 

software quickly storing the raw image data in a first memory, wherein the 

The NTSC to PAL Resize block 418 generates 720 tost memory contains raw image data for one or more 

(CCIR) or 768 (Square Pixel) horizontal pixels from the 65 images; 

incoming 640 NTSC pixels when PAL mode is selected. accessing a particular raw image dala from the first 

Also, six lines are generated for every five incoming lines memory and performing image processing on the par- 
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licular raw image data through an image processing 
data path to produce a rotated processed image, 
wherein the image processing data path comprises edge 
enhancement, interpolation, ring insertion, pixel 
shuffle, line averaging, and decompression; 

loading the processed image in a second buffer; 

compressing the processed image to generate a com- 
pressed image; 

unrotating the image; 

performing display processing to produce graphics for 
display; 

storing the graphics and image in a fourth buffer; and 
rendering the graphics and image from the fourth buffer 
for display on the image capture unit. 

3. In a digital camera, a method for digital image 
processing, comprising the steps of: 

generating a live view image for display with the digital 
camera in a horizontal, vertical, or upside-down orien- 
tation; 

capturing an image for subsequent display, wherein the 
image is captured with the digital camera in either a 
horizontal or vertical orientation; 

rendering a captured image for instant review with the ^ 
digital camera in either the horizontal or vertical ori- 
entation and with the captured image rotated or not 
rotated; 

rendering the captured image for a play mode of operation 
with the digital camera in either the horizontal or 
vertical orientation and with the captured image rotated 
or not rotated; 

rotating the image when the digital camera is in the 
vertical orientation to produce a horizontal image; 

storing the horizontal image in memory; 

rotating the horizontal image from memory for displaying 
the live view; 

resizing the horizontal image for instant review or play 
mode if the digital camera is in a horizontal orientation; 

displaying a resized horizontal image with a pre- 
determined background on either side of the resized 
horizontal image; 

displaying an indication that the horizontal image was 
taken when the digital camera was in the vertical 45 
orientation; 

rotating the horizontal image in a second pass if the digital 
camera is physically rotated to the vertical orientation. 

4. A digital signal processor for use in an image capture 
apparatus, comprising: 

a bus 

a capture data path coupled to the bus for accepting CCD 
data and outputting raw lines to the bus, wherein the 
capture data path includes a statistics circuit, a pixel 
average circuit, and a compression circuit; 

a memory coupled to the bus for storing the raw lines; 

a data flow controller coupled to the bus for generating 
raw minimum computational units (MCUs) based on 
the raw lines stored in the memory; 

an image processing data path for processing the raw 
MCUs from the data flow controller; 

a buffer for storing the MCU's from the image processing 
data path; 
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a resize circuit coupled to the buffer for resizing image 
data; 

a display processor coupled to the resize circuit for 
generating graphical information for display with the 
image data; 

a rotation circuit coupled to the bus for rotating the image 
data and storing a rotated image data in the buffer. 

5. A digital signal processor for use in an image capture 
apparatus, comprising: 

a bus 

a capture data path coupled to the bus for accepting CCD 

data and outputting raw lines to the bus; 
a memory coupled to the bus for storing the raw lines; 
a data flow controller coupled to the bus for generating 

raw minimum computational units (MCUs) based on 

the raw lines stored in the memory; 
an image processing data path for processing the raw 

MCUs from the data flow controller; 
a buffer for storing the MCU's from the image processing 

data path; 

a JPEG engine for compressing the MCU's before storage 
in the memory and for decompressing MCU's read 
from the memory; 

a resize circuit coupled to the buffer for resizing image 
data; 

a display processor coupled to the resize circuit for 
generating graphical information for display with the 
image data; 

a rotation circuit coupled to the bus for rotating the image 
data and storing a rotated image data in the buffer, 

wherein at least four modes of operation are supported: 
live view, image capture, instant review, and play 
mode, and wherein the image is automatically rotated 
in a second pass in accordance with a current physical 
orientation of the image capture apparatus and with the 
physical orientation of the image capture apparatus 
when the image was captured. 

6. The digital signal processor of claim 5, wherein in the 
live view mode of operation, the image from the capture data 
path is stored in an input buffer of the memory, retrieved and 
processed by the image processing path, buffered, resized if 
necessary, and displayed along with any appropriate graph- 
ics. 

7. The digital signal processor of claim 6, wherein if the 
image capture apparatus is in a vertical orientation, the 
image is rotated by the rotation circuit before being stored in 
the memory and the image is rotated again before being 
rendered for display. 

8. The digital signal processor of claim 5, wherein in a 
capture mode of operation, the image from the capture data 
path is stored in an input buffer of the memory, retrieved and 
processed by the image processing path, buffered, com- 
pressed by the compression/decompression engine, and 
stored in a file of the memory. 

9. The digital signal processor of claim 8, wherein if the 
image capture apparatus is currently in a horizontal orien- 
tation and the image was taken while the image capture 
apparatus was in a vertical orientation, the image is reduced 
in size before being rendered for display. 

10. The digital signal processor of claim 9, wherein the 
display processor generates an indication to the user to 
change the orientation of the image capture apparatus and 
when the image capture apparatus is rotated to be in the 
vertical orientation, the image is rendered for display wilh- 



a JPEG engine for compressing the MCU's before storage 65 out resizing, 
in the memory and for decompressing MCU's read 11. The image capture apparatus of claim 5, wherein in an 
from the memory; instant playback mode of operation, the image from the 
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capture data path is stored in an input buffer of the memory, 
retrieved and processed by the image processing path, 
buffered, compressed by the compression/decompression 
engine, stored in a file of the memory, read back from the 
file, decompressed, resized if necessary, and displayed along 5 
with any appropriate graphics. 

12. The image capture apparatus of claim 5 further 
comprising an orientation detector for detecting a current 
physical orientation of the image capture apparatus, wherein 
the image from a first draw buffer is rotated by the rotation 
circuit, depending on the current orientation, and stored in a 
second draw buffer. 

13. The image capture apparatus of claim 5, wherein in a 
playback mode of operation, one or more requested files are 
read, decompressed, resized if necessary, and displayed 
along with any appropriate graphics. 15 

14. The image capture apparatus of claim 5 further 
comprising an orientation detector for detecting a current 
physical orientation of the image capture apparatus, wherein 
the image from a first draw buffer is rotated by the rotation 
circuit, depending on the current orientation, stored in a 20 
second draw buffer, and displayed in a correct orientation. 

15. A digital signal processor for use in an image capture 
apparatus, comprising: 

a bus 

a capture data path coupled to the bus for accepting CCD 25 

data and outputting raw lines to the bus; 
a memory coupled to the bus for storing the raw lines; 
a data flow controller coupled to the bus for generating 

raw minimum computational units (MCUs) based on 

the raw lines stored in the memory; 30 
an image processing data path for processing the raw 

MCUs from the data flow controller; 
a buffer for storing the MCU's from the image processing 

data path; 35 
a JPEG engine for compressing the MCU's before storage 

in the memory and for decompressing MCU's read 

from the memory; 
a resize circuit coupled to the buffer for resizing image 

data; ^ 
a display processor coupled to the resize circuit for 

generating graphical information for display with the 

image data; 

a rotation circuit coupled to the bus for rotating the image 
data and storing a rotated image data in the buffer; 45 

a double buffered register, wherein transfers of data 
between buffers is activated at frame boundaries; 

wherein at least four modes of operation are supported: 
live view, image capture, instant review, and play 
mode. 50 

16. The image capture apparatus of claim 5 further 
comprises a YCC shuffle block having a data path coupled 
directly to the buffer. 

17. A method for processing an image in an image capture 
unit including a display, the method comprising the steps of: 55 

capturing raw image data with an image sensor; 
processing the raw image data through a capture data 
path; 

transferring the raw image data from the image sensor to 
a first memory via a bus for storage, wherein the first 60 
memory contains raw image data for one or more 
images; 

transferring a particular raw image data from the first 
memory to a data flow control via the bus; 

passing the raw image data from the data flow control to 65 
an image processing data path, the image processing 



535 Bl 

20 

data path for performing image processing on the 
particular raw image data to produce a rotated pro- 
cessed image; 

transferring the processed image from the image process- 
ing data path to a second buffer through a first dedicated 
path; 

establishing a data path between the second buffer an a 
compression/decompression engine; 

compressing the processed image to generate a com- 
pressed image; 

transferring the compressed image from the compression/ 
decompression engine to a third buffer via the bus to 
store a compressed displayable image; 

retrieving the compressed displayable image from the 
third buffer for decompression by the compression/ 
decompression engine to produce displayable image 
data; 

transferring the displayable image data to the second 
buffer; 

transferring the displayable image data from the second 
buffer to a resize circuit through a second dedicated 
path; 

performing display processing to produce graphics for 
display; 

transferring the graphics and displayable image to a fourth 

buffer via the bus; 
loading the graphics and displayable image from the 

fourth buffer to a rotate circuit through the bus; 
unrotating the displayable image; 
rendering the graphics and displayable image from the 

fourth buffer for display on the image capture unit; and 
establishing a third dedicated path from the rotate circuit 

to the second buffer. 
18. A method for processing an image in an image capture 
unit including a display, the method comprising the steps of: 
capturing raw image data; 

processing the raw image data through a capture data 
path; 

storing the raw image data in a first memory, wherein the 
first memory contains raw image data for one or more 
images; 

accessing a particular raw image data from the first 
memory and performing image processing on the par- 
ticular raw image data to produce a rotated processed 
image; 

loading the processed image in a second buffer; 
compressing the processed image to generate a com- 
pressed image; 
unrotating the image; 

performing display processing to produce graphics for 
display; 

storing the graphics and image in a fourth buffer; 
rendering the graphics and image from the fourth buffer 

for display on the image capture unit; and 
performing a second pass when the image capture unit is 

rotated, the second pass comprising the steps of: 

accessing the image from the fourth buffer; 

rotating the image; 

storing the image in the second buffer; 
passing the image through a resize circuit; 
adding the graphics during the second pass; and 
storing the image in a fifth buffer. 

* * * * ♦ 
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